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EXECUTIVE  SUMMARY 

The  E&V  Project  and  the  E&  V  Team  were  formed  in  1 983 .  The  purpose  of  the  project  was  to  pro¬ 
vide  a  capability  to  assess  APSEs  and  to  determine  conformance  of  APSEs  to  applicable  standards.  The 
purpose  of  the  team  was  to  assist  the  project  in  several  ways.  Raymond  Szymanski  of  Wright  Research 
and  Development  Center  (WRDC,  now  Wright  Laboratory,  WL)  served  as  project  and  team  leader  from 
1983  until  the  completion  of  team  activities,  and  brought  continuity  that  provided  a  setting  for  produc¬ 
tive  contractual  efforts  and  coordinated  team  products*. 

The  principal  products  of  the  contractual  efforts  are:  a  test  suite  known  as  the  Ada  Compiler 
Evaluation  Capability  (ACEC),  a  pair  of  documents  known  as  the  E&V  Reference  System,  and  a  test 
suite  known  as  die  CAIS  Implementation  Validation  Capability  (CTVC).  The  E&V  Team  held  quarterly 
meetings  that  produced  many  suggestions  and  a  number  of  documents  which  had  a  significant  influence 
on  die  contractor-developed  products.  A  report  from  die  team  chairman  summarizing  die  team’s  pur¬ 
pose,  process,  and  products,  as  well  as  his  personal  prognosis  and  recommendations  is  included  in  this 
report  (Section  2).  He  concludes  that  considerable  work  is  still  required  to  enable  DoD  to  derive  full 
benefit  from  the  E&V  effort,  and  recommends  specific  steps  to  transition  and  use  DoD-developed  E&V 
technology  and  to  promote  the  development  of  additional  E&V  technology.  Summary  reports  from  each 
of  the  team’s  working  groups  are  also  included  (Section  3).  The  list  below  summarizes  the  team’s  recom¬ 
mendations  concerning  future  use  and  development  of  E&V  technology.  A  more  complete  statement  is 
provided  in  Section  4. 

E&V  Team  Recommendations 

1.  Raise  public  awareness  of  E&V  and,  as  a  broad  objective,  encourage  use  of  APSE 
E&V  technology 

2.  Mandate  compiler  evaluations 

3.  Create  a  Quality  Testing  Center  Of  Expertise 

4.  Continue  development  of  E&V  technology  in  four  areas  and  coordination  with  re¬ 
lated  activities 

3.  Promote  applicable  APSE  standards 

6.  Identify  an  appropriate  organization  for  APSE  E&V  standards  activities;  develop, 
manage,  and  maintain  E&V  test  suites  using  a  unified  approach 

7.  Promote  international  sharing  of  E&V  technology,  but  avoid  joint  multinational  de¬ 
velopment  of  assessor  products. 

*  Virginia  Castor  served  as  project  and  team  leader  from  December  1983  to  June  1983. 
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1.  INTRODUCTION 

1.1  PURPOSE  OF  THIS  DOCUMENT 

This  report  has  been  prepared  to  reflect  the  current  state  of  Ada  Programming  Support  Environ¬ 
ment  (APSE)  Evaluation  and  Validation  (E&V)  technology  as  developed  by  the  E&V  Project,  and  the 
recommendations  of  the  E&V  Team  regarding  potential  future  directions.  The  E&V  Team  has  com¬ 
pleted  its  charter  and  has  prepared  this  report  for  reader  consideration.  The  final  team  recommendations 
are  given  in  Section  4.  This  report  does  not  necessarily  reflect  the  opinion  of  the  Ada  Joint  Program 
Office  (AJPO),  the  U.S.  Air  Force,  or  the  Wright  Research  and  Development  Center  (WRDC).  Rather,  it 
represents  the  participants’  perspectives  regarding  future  directions  for  the  E&V  Reference  System,  the 
Ada  Compiler  Evaluation  Capability  (ACEC),  the  Common  APSE  Interface  Set  (CAIS)  Implementa¬ 
tion  Validation  Capability  (CIVC),  and  APSE  E&V,  in  general. 


1.2  BACKGROUND 

The  AJPO  was  formed  in  December  1980  by  the  Under  Secretary  of  Defense  for  Research  and 
Engineering  to  manage  the  efforts  related  to  the  introduction,  implementation,  and  life  cycle  support  of 
the  Ada  programming  language.  One  of  these  efforts  is  the  development  of  APSE  E&V  technology.  The 
AJPO  is  responsible  for  ensuring  that  the  Department  of  Defense  (DoD)  has  the  programming  support 
tools  needed  to  develop  and  support  defense  systems  software  written  in  the  Ada  language,  and  that  these 
tools  conform  to  DoD  standards. 

In  June  1983,  the  AJPO  proposed  the  formation  of  the  E&V  Project  and  a  tri-service  APSE  E&V 
Team  with  die  Air  Force  designated  as  lead  service.  In  October  1983;  the  Air  Force  officially  accepted 
responsibility  as  lead  service  on  the  E&V  Project  with  the  Air  Force  Wright  Aeronautical  Laboratory 
( AFWAL,  now  known  as  the  Wright  Research  and  Development  Center)  as  die  lead  organization  and  die 
Avionics  Laboratory  as  the  lead  laboratory  for  the  project.  The  project  was  tasked  with  the  following:  ( 1 ) 
identifying  and  defining  specific  E&V  technology  requirements,  (2)  developing  selected  elements  of 
the  required  technology,  (3)  encouraging  others  to  develop  additional  elements,  and  (4)  collecting 
information  describing  existing  elements. 
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i-J  TEAM  ORGANIZATION 

The  project’s  technical  team  was  formed  in  December  1983  and  was  designated  the  E&V  Team. 
The  E&V  Team  was  charged  with  the  responsibility  of  developing  a  variety  of  foundation  documents 
and  providing  technical  guidance  early-on  in  the  E&V  Project.  The  E&V  Team  was  a  Do D  team  with 
members  from  the  three  services  and  other  DoD  agencies. 

In  April  1984,  following  a  precedent  set  by  another  DoD  tri-service  team,  the  KAPSE  Interface 
Team  (KIT),  the  E&V  Project  invited  several  representatives  from  industry  and  academia  to  participate 
in  a  workshop  [E&V  Workshop  1984].  These  representatives  became  Distinguished  Reviewers  (DRs), 
who  continued  to  supplement  die  team  knowledge  base  and  provided  the  E&V  Team  with  a  broad  range 
of  inputs,  reviews,  and  advice  of  the  highest  technical  quality. 

The  E&V  Team  was  initially  partitioned  into  several  working  groups  which  produced  white 
papers  and  other  reports  on  E&V  topics.  The  charter  for  each  of  these  groups  and  the  work  they 
performed  are  recorded  in  a  series  of  E&V  Team  Public  Reports.  [E&V  Report  1984, 1985, 1987, 1989, 
1990],  As  time  progressed,  however,  die  E&V  Project  began  to  award  contracts  for  the  development  of 
E&V  technology.  In  order  to  exploit  the  technical  expertise  oft he  E&V  Team  for  die  benefit  of  the  tech¬ 
nical  developments,  the  team  was  reorganized  into  die  following  six  working  groups: 

•  ACECWG  —  The  Ada  Compiler  Evaluation  Capability  (ACEC)  Working  Group 
was  responsible  for  providing  technical  inputs  to  the  ACEC  product. 

•  CTVCWG  —  The  CAIS  Implementation  Validation  Capability  (CIVC)  Working 
Group  was  responsible  for  providing  technical  inputs  to  the  CIVC  product 

•  CLASS WG  —  The  Classification  Working  Group  was  responsible  for  providing 
technical  inputs  to  die  E&V  Reference  System  documents. 

•  PUBWG  —  The  Publicity  Working  Group  was  responsible  for  assisting  in  the 
development  and  review  of  E&V  Project  publicity  information  and  E&V  Team 
Public  Reports.  (This  group  later  merged  into  the  REQWG;  therefore,  there  is  no 
PUBWG  report  in  this  document) 

•  REQWG  —  The  Requirements  Working  Group  was  responsible  for  E&V  Project 
requirements. 

•  SEVWG — The  Standards  Evaluation  and  Validation  Working  Group  was  respon¬ 
sible  for  reviewing  APSE  related  standards  such  as  MIL- STD- 1 838 A,  the  CAIS-A 
standard,  [DoD  1989]  to  identify  E&V  technology  needs. 

These  groups  worked  together  to  ensure  that  die  E&V  Project  developments  met  die  needs  of  the 
DoD  user  community.  The  accomplishments  and  successes  of  these  working  groups  are  detailed  in 
upcoming  sections. 
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1.4  DOCUMENT  ORGANIZATION 


Section  1  —  Introduction,  provides  the  purpose  of  this  report  as  well  as  background  for  the  exis¬ 
tence  of  the  E&V  Team  and  its  associated  working  groups. 

Section  2  —  Report  of  the  E&V  Team  Chairman  to  the  AJPO,  presents  a  summary  of  the  E&V 
Team  activities  and  products  since  the  team ’s  inception  in  December  1983 ,  and  the  chairman’s  prognosis 
and  recommendations. 

Section  3 — Working  Group  Reports,  presents  the  final  reports  of  die  working  groups  along  with 
recommendations  specific  to  individual  working  groups. 

Section  4 — Recommendations,  presents  the  major  recommendations  of  the  E&V  Team. 

Section  5  —  Conclusion,  provides  a  brief  concluding  statement. 

Appendices  A,  B,  and  C  define  acronyms,  list  team  participants,  and  cite  references, 
respectively. 
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2.  REPORT  OF  THE  E&V  CHAIRMAN  TO  THE  AJPO 


2.1  THE  PURPOSE  OF  THE  E&V  TEAM 

In  October  1983,  the  Air  Force  officially  accepted  responsibility  as  lead  service  on  the  E&V 
Project.  The  purpose  of  the  E&V  Project  was  to  provide  a  capability  to  assess  APSEs  and  to  determine 
conformance  of  APSEs  to  applicable  standards.  Figure  2.1-1  provides  a  pictorial  view  of  the  intended 
role  of  E&V  technology  in  DoD  system  developments.  The  E&V  Team  was  formed  to  assist  the  E&V 
Project  as  follows: 

•  Develop  an  E&V  Requirements  Document  against  which  E&V  Project  products 
and  Team  activities  could  be  assessed. 

•  Provide  recommendations  for  development/acquisition  of  E&V  tools/aids  through 
the  development  of  an  E&V  Tools  and  Aids  document. 

•  Provide  technical  guidance  in  early  stages  of  E&V  Project  product  developments. 


Figure  2.1-1  The  Role  of  E&V  Technology 
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The  E&V  Requirements  Document,  the  E&V  Tools  and  Aids  Document,  and  the  Issues  and 
Strategies  for  CALS  Evaluation  and  Validation  Document  have  all  been  completed  [E&V  Requirements 
1987]  [E&V  Tools  and  Aids  1990]  [Issues  1990].  Additionally,  each  of  the  E&V  Project  product  devel¬ 
opments  has  produced,  as  a  minimum,  two  versions  of  their  respective  products  (E&V  Reference 
System,  ACEC,  and  CTVC).  These  products  were  substantially  influenced  by  the  E&V  Team. 


2.2  THE  PROCESS 

The  E&V  Team  met  quarterly  from  December  1983  through  September  1990  for  a  total  of  28 
meetings.  The  results  of  these  meetings  are  a  number  of  E&V  technology  developments  which  are  in  use 
today  by  the  DoD.  Although  the  E&V  Project  can  be  considered  successful  for  its  present  accomplish¬ 
ments,  considerable  work  is  still  required  to  enable  the  DoD  to  derive  full  benefit  from  the  E&V  effort. 

Now  that  the  work  of  the  E&V  Team  has  reached  completion,  it  is  time  to  consider  the  future  of 
E&V  technology.  As  reflected  in  the  recommendations  that  follow,  there  is  a  need  to  enhance  the 
existing  E&V  Project  products  and  to  undertake  new  E&V  technology  developments.  Additionally,  the 
DoD  should  take  steps  to  encourage  the  use  of  available  E&V  technology.  This  encouragement  could  be 
manifested  through  changes  in  acquisition  policy  which  would  require  the  use  of  E&V  technology  prior 
to  acquisition  of  critical  software  tools.  Finally,  die  team  believes  that  “centers  of  expertise”  should  be 
established  by  the  DoD  for  the  application,  enhancement,  and  development  of  E&V  technology. 


23  THE  PRODUCTS  —  PRODUCED  BY  THE  E&V  TEAM 

2.3.1  Requirements  Document  [E&V  Requirements  1987] 

This  document  sets  forth  the  requirements  on  the  E&V  Project.  It  was  intended  for  use  by  the 
E&V  Team  when  determining  which  activities  to  pursue  and  by  the  support  contractors  when  develop¬ 
ing  technology  for  the  E&V  Project. 

232  Component  Validation  Procedures  Document  [CVP 1985] 

This  document  examined  procedures  for  validating  APSE  components  for  which  a  standard 
exists.  It  proposed  a  process  for  assuring  consistent  implementations  of  the  CAIS  through  administra¬ 
tion  of  a  validation  capability  that  is  similar  to  the  procedures  adopted  for  Ada  language  validation. 
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2JJ  Issues  and  Strategies  for  CAIS  Evaluation  and  Validation  [Issues  1987] 

This  document  was  developed  during  the  later  phases  of  the  DoD-STD-1 838  (CAIS)  develop¬ 
ment  cycle  and  it  made  significant  recommendations  for  modifications  to  the  proposed  CAIS  standard. 
One  particular  recommendation  included  the  addition  of  a  package  to  facilitate  validation  test  suites  and 
automatic  adaptation  of  tools.  Other  recommendations  focused  on  improvements  to  the  readability  and 
understandability  of  the  document  defining  the  standard. 

13.4  Issues  and  Strategies  for  Evaluation  and  Validation  of  CAIS-A 
Implementations  [Issues  1990] 

Based  on  the  CAIS-A  standard  (MIL-STD-1838A)  and  results  of  the  CTVC  project,  this  docu¬ 
ment  covers  topics  relevant  to  creating  a  validation  mechanism  for  CAIS-A.  The  document  raises  issues 
relating  to  cost,  test  selection  criteria,  maintenance  of  the  validation  suite,  and  tools  to  aid  in  construction 
of  the  validation  suite.  Recommendations  contained  in  the  document  suggest  ways  to  resolve  the  issues 
discussed. 

233  Tools  and  Aids  Document  [E&V  Tools  and  Aids  1990] 

This  document  includes  recommendations  for  specific  elements  of  E&V  technology  which  need 
to  be  developed,  and  an  accompanying  prioritization  of  their  implementation  order.  Appendices  of  the 
Tools  and  Aids  Document  provide  detailed  specifications  for  selected  recommended  elements. 

2.4  THE  PRODUCTS  —  WITH  THE  E&V  TEAM  AS  TECHNICAL  CONTRIBUTORS 

2.43  Ada  Compiler  Evaluation  Capability  (ACEC) 

The  Ada  Compiler  Evaluation  Capability  (ACEC)  is  being  developed  by  Boeing  Military  Air¬ 
plane  under  contract  to  the  Air  Force  Wright  Research  and  Development  Center  (WRDC)  with  funding 
from  the  AJPO.  Its  primary  purpose  is  to  provide  the  capability  to  determine  the  performance  and  usabil¬ 
ity  characteristics  of  Ada  compilation  systems.  The  ACEC  consists  of  the  ACEC  Software  Product  and 
three  supporting  documents:  the  ACEC  User’s  Guide,  the  ACEC  Reader’s  Guide,  and  die  ACEC 
Version  Description  Document  [ACEC  1990]  [ACEC  1990a]  [ACEC  1990b]. 

2.43.1  ACEC  Software  Product  —  The  ACEC  Software  Product  consists  of  performance 
tests,  assessor  tools,  and  support  software.  The  software  product  makes  it  possible  to: 

•  Compare  the  performance  of  several  implementations. 

•  Isolate  the  strong  and  weak  points  of  a  specific  system,  relative  to  other  systems 
which  have  been  tested. 
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•  Determine  what  significant  changes  were  made  between  releases  of  a  compilation 
system. 

•  Predict  performance  of  alternative  coding  styles. 

•  Evaluate  the  clarity  and  accuracy  of  a  system’s  diagnostic  messages. 

•  Determine  whether  the  functional  capabilities  of  a  program  library  system  are  suffi¬ 
cient  to  accomplish  a  set  of  predefined  scenarios. 

•  Determine  whether  the  functional  capabilities  of  a  symbolic  debugger  are  sufficient 
to  accomplish  a  set  of  predefined  scenarios. 

The  ACEC  performance  tests  provide  assistance  in  measuring  execution  time  efficiency,  code 
size  efficiency,  and  compile  time  efficiency.  The  assessor  tools  provide  assistance  in  evaluating 
symbolic  debuggers,  program  library  systems,  and  compiler  diagnostics.  The  test  suite  does  not  include 
explicit  tests  for  the  existence  of  language  features. 

The  support  software  is  a  set  of  tools  and  procedures  which  assist  in  preparing  and  executing  the 
test  suite ,  in  extracting  data  from  the  results  of  executing  the  test  suite,  and  in  analyzing  the  performance 
measurements  obtained.  The  support  tools  consist  of: 

•  INCLUDE  —  assists  in  adapting  programs  to  particular  targets  by  performing 
source  text  inclusion. 

•  FORMAT  and  MED_D ATA_CONSTRU CTOR  —  extract  and  format  timing  and 
code  expansion  data. 

•  MEDIAN  —  compares  results  of  performance  tests  of  various  systems. 

•  SINGLE  SYSTEM  ANALYSIS  —  compares  results  of  related  tests  from  a  single 
system. 

The  /  .CEC  Software  Product  was  developed  for  a  variety  of  targets  and  is  distributed  on  one 
9-track,  1600  bpi,  VAX/VMS  backup  tape  containing  approximately  20  megabytes  of  data. 

2.4.I.2  ACEC  Documentation  Products  —  The  ACEC  Documentation  Products  are  as 
follows: 

•  ACEC  User’s  Guide  —  The  ACEC  User’s  Guide  [ACEC  1990a]  provides  ACEC 
users  with  the  information  necessary  to  adapt  and  execute  the  ACEC  Software 
Product.  This  guide  explains  how  to  use  the  support  tools  and  how  to  deal  with  the 
problems  which  may  occur  in  the  process  of  adapting  and  executing  the  ACEC 
Software  Product. 

•  ACEC  Reader’s  Guide  — The  ACEC  Reader’s  Guide  [ACEC  1990]  describes  how 
users  can  interpret  the  results  of  executing  the  performance  tests  and  assessor  tools. 
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This  guide  also  discusses  the  statistical  significance  of  the  numbers  produced,  die 
organization  of  the  test  suite,  and  the  submission  of  error  reports  and  change 
requests. 

•  ACEC  Version  Description  Document  —  The  Version  Description  Document 
[ACEC  1990b]  describes  the  ACEC  Software  Product  as  contained  on  the  distribu¬ 
tion  tape,  including  the  compilation  units,  programs,  test  problems,  and  sample 
data.  This  document  also  contains  a  set  of  indexes  which  allow  the  user  to  identify 
each  test,  its  primary  purpose,  its  secondary  and  incidental  purposes,  related  and 
comparison  test  problems,  and  applicable  Ada  Language  Reference  Manual  [DoD 
1983]  sections. 

2.4i  CAIS  Implementation  Validation  Capability  (CIVC) 

The  CAIS  Implementation  Validation  Capability  (CTVC)  is  being  developed  by  SofTech  under 
contract  to  the  Air  Force  Wright  Research  and  Development  Center  (WRDC)  with  funding  from  the 
AJPO.  The  CTVC  effort  is  developing  a  validation  test  suite  to  assess  conformance  of  CAIS  implementa¬ 
tions  to  MEL-STD-1838A  [DoD  1989].  The  CAIS  is  an  extensive  set  of  interfaces  designed  to  support 
the  development  of  portable  tools  for  use  in  APSEs.  The  CIVC  has  successfully  applied  information 
modeling  to  the  test  coverage  design  and  assessment  required  for  validation  testing.  Hypermedia  has 
been  used  for  the  delivery  of  test  requirements,  test  designs,  and  their  associated  traceability  relation¬ 
ships.  Version  1  of  die  CIVC  (CTVC1)  was  delivered  in  February  1990,  and  assesses  conformance  to 
DoD-STD-1838  [DoD  1986].  Version  2  of  the  CIVC  (CTVC2)  is  being  prepared  for  delivery  in  February 
1992,  and  will  be  used  to  assess  conformance  to  MEL-STD-1838A. 

2.42.1  CIVC  Software  Products  —  The  software  components  of  the  CTVC  are  the  Frame¬ 
work,  the  Test  Administrator,  and  the  Test  Suite.  Each  of  these  components  is  briefly  described  below. 

•  CTVC1  Framework  —  The  CTVC1  Framework  is  a  hypertext-based  product  that 
provides  complete  and  unique  traceability  between  DoD-STD-1838  and  the 
CTVC1  software  product.  The  Framework  product  provides  the  means  for 
evaluating  die  correctness  of  both  the  CTVC1  product  and  proposed  additions  to  the 
test  suite.  See  [CTVC  1990]. 

•  Test  Administrator — The  Test  Administratorprovides  the  CTVC1  and  CTVC2  Beta 
release  user  interfaces,  encapsulates  any  target  environment  dependencies  for  oper¬ 
ating  these  suites,  ami  schedules  and  executes  the  CTVC1  and  CTVC2  validation 
tests  defined  in  the  Test  Suite.  See  [CIVC  1990a]. 

•  CTVC1  Test  Suite — Test  objectives,  scenarios,  test  cases,  static  compilation  tests, 
and  a  report  manager  constitute  the  CTVC1  Test  Suite.  The  Test  Suite  encapsulates 
the  actual  tests  which  exercise  a  CAIS  implementation  and  is  organized  by  super¬ 
classes.  Superclasses  are  arbitrary  organizations  of  test  classes.  Test  classes  are 
groups  of  test  cases  which  either  have  similar  preconditions,  or  have  preconditions 
which  depend  on  the  postcondition  of  a  previously  executed  test  case. 
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•  CIVC2  Beta  Test  Suite — A  preliminary  validation  capability  of  M1L-STD- 1838 A 
was  delivered  in  July  1990  as  the  CTVC2  Beta  Test  Suite.  Derived  from  the  CTVCl 
Test  Suite,  147  test  cases  were  modified  to  exercise  the  CAIS-A  interfaces. 

•  CIVC2  Framework  —  To  support  the  evaluation  for  correctness  of  the  CIVC2, 
completeness  and  consistency  will  be  determined  by  developing  a  full  traceability 
framework  between  the  requirements  of  MIL- STD- 1838 A  and  the  CTVC2  software 
product.  The  CIVC2  Framework  will  incorporate  aspects  of  the  latest  hypertext 
technology  (i.e.,  “sticky-text”  links). 

•  CIVC2  Test  Suite  — Test  objectives,  scenarios,  test  cases,  static  compilation  tests, 
and  a  report  manager  constitute  the  CIV C2  Test  Suite.  The  Test  Suite  encapsulates 
the  actual  tests  and  is  organized  by  test  classes.  Test  selection  is  accomplished  by 
automated  analysis  and  prioritization  systems.  See  [CTVC  1990b]. 

Collectively,  these  products  (1)  discover  and  report  ambiguities,  incomplete  parts,  and  other 
potential  impediments  to  common  interpretations  of  CAIS/CAIS-A,  and  (2)  produce  mechanisms  for 
analyzing  and  reporting  errors  in  CAIS/CAIS-A  implementations  that  violate  specifications. 

2. 4.2. 2  CTVC  Documentation  Products — The  CTVC  Documentation  Products  are  as  follows: 

•  CTVCl  Implementor’s  Guide  —  The  CTVCl  Implementor’s  Guide  [CTVC  1989] 
presents  the  conformance  requirements  (test  objectives)  for  a  CAIS  implementa¬ 
tion,  as  well  as  the  top  level  designs  (scenarios)  for  die  test  cases  that  will  validate 
a  CAIS  implementation’s  conformance  to  DoD-STD-1838.  Also  included  are  die 
conventions  and  rationale  used  in  die  development  of  these  CTVCl  products. 

•  CTVCl  Framework  —  The  hypertext-based  framework  provides  the  traceability 
documentation  between  the  DoD-STD-1838  and  the  CTVCl  product.  See 
[CTVC  1990]. 

•  Test  Report  Reader’s  Guide  with  Appendix  1- Operator’s  Guide — [CTVC  1990a] 

This  Technical  Operating  Report  describes  the  format  of  the  CTVC  Conformance 
Repon  and  how  to  interpret  the  data  presented  in  the  Conformance  Report  The 
Operator’s  Guide  details  the  system  requirements,  installation,  and  operation  of 
CIVC1. 

•  CTVC2  Beta  Test  Suite  Operator’s  Guide  [CTVC  1990b] — This  document  details 
the  host  system  requirements,  installation,  and  operation  of  the  CTVC2  beta  release 
(CIVC-Beta).  This  beta  release  of  the  CTVC  provides  early  validation  support  to 
CAIS-A  implementors. 

The  CTVC2  documentation  products  corresponding  to  the  CTVCl  documentation  products 
above  will  be  delivered  in  early  1992. 
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2.4 3  Evaluation  and  Validation  Reference  System 

The  Evaluation  and  Validation  Reference  System  is  being  developed  by  TASC  under  contract  to 
the  Air  Force  Wright  Research  and  Development  Center  (WRDC)  with  funding  from  the  AJPO.  It 
consists  of  two  companion  documents:  the  E&V  Reference  Manual,  Version  2.0  [E&V  RM  1989]  and 
the  E&V  Guidebook,  Version  2.0  [E&V  GB  1989].  Version  3.0  of  both  documents  will  be  available  in 
early  1991. 

The  purpose  of  the  E&V  Reference  Manual  is  to  provide  information  that  will  help  users  to: 

•  Gain  an  overall  understanding  of  APSEs  and  approaches  to  their  assessment. 

•  Find  useful  reference  information  (e.g.,  definitions)  about  specific  elements  and 
relationships  between  elements. 

•  Find  criteria  and  metrics  for  assessing  tools  and  APSEs,  and  techniques  for 
performing  such  assessments. 

Chapters  1  through  3  provide  a  general  introduction  to  the  document  and  to  die  issue  of  assessing 
APSEs  as  a  whole.  Chapter  4  and  later  chapters  each  correspond  to  one  index  of  an  overall  E&V  Classifi¬ 
cation  Schema.  The  schema  adopts  a  relational  model  of  the  subject  and  process  of  E&V.  This  model 
allows  the  user  to  arrive  at  E&V  techniques  through  many  different  paths,  and  provides  a  means  to 
extract  useful  information  along  the  way. 

The  purpose  of  the  E&V  Guidebook  is  to  provide  information  that  will  help  users  to  assess 
APSEs  and  APSE  components  by: 

•  Assisting  in  the  selection  of  E&V  procedures,  the  interpretation  of  results,  and  the 
integration  of  analyses  and  results. 

•  Describing  E&V  procedures  and  techniques  developed  by  the  E&V  Project. 

•  Assisting  in  the  location  of  EAV  procedures  and  techniques  developed  outside  the 
E&V  Project. 

All  E&V  procedures  and  techniques  found  in  the  Guidebook  are  referenced  by  the  indexes 
contained  in  the  E&V  Reference  Manual.  Chapters  1  through  4  provide  a  general  introduction  to  the  doc¬ 
ument  and  other  background  material.  Chapter  5  and  later  chapters  each  contain  all  the  assessment 
procedures  and  techniques  associated  with  a  particular  group  of  tools  or  tool  sets  to  be  assessed,  such  as 
Compilation  System  Assessors  or  Test  System  Assessors.  The  assessment  procedures  are  described  and, 
in  some  instances,  can  be  applied  directly  from  the  information  given  in  die  Guidebook.  In  other  cases, 
the  user  is  directed  to  a  primary  reference  for  more  information.  Figure  2.4.3- 1  illustrates  the  relation¬ 
ship  between  the  Reference  Manual  and  the  Guidebook. 
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(1)  Useful 

Information 
Directly  from 
the  Manual 


Users  May  Consult 
the  Reference 
Manual  to  Extract 


or 


o-i  mi 
9-T-40 


Directly  Consult 
the  Guidebook 


or  (2)  Pointers  to 


Sections  In 
the  Guidebook 


—Whfeh  Provides  Information  About 
E&V  Tools  and  Techniques 


Figure  2.43-1  The  E&V  Reference  System 


The  contractor  has  also  begun  development  of  a  set  of  structured  experiments  designed  to  evalu¬ 
ate  integrated  APSEs.  These  are  built  around  a  “model  project”  testbed  that  has  been  partially  designed, 
implemented,  tested,  and  documented  using  DoD-STD-2167A  [DoD  1988]  formats.  The  initial  set  of 
experiments  consist  of  scripted  scenarios,  which  emphasize  software  testing  activities.  Rather  than 
focus  on  the  evaluation  of  a  single  test  tool  or  function,  die  scenarios  are  designed  with  a  “whole  team” 
perspective  and  address  a  spectrum  of  activities  such  as  test  planning,  documentation,  unit  testing,  com¬ 
ponent  testing,  integration  testing,  bug  fixing,  regression  testing,  status  repotting,  and  configuration 
control.  The  experiments  are  designed  to  provide  an  initial  baseline  that  can  be  extended  to  cover  other 
aspects  of  “whole  APSE”  performance  and  “whole  team”  support. 
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23  THE  PROGNOSIS 

It  is  my  belief  that  the  E&V  Team  effort  has  been  successful.  Although  its  activities  were 
extremely  diverse,  it  has  produced  documents  which  should  guide  the  DoD  in  its  future  E&V  endeavors. 
Additionally,  the  Team  has  made  outstanding  technical  contributions  to  the  E&V  Project’s  technical 
developments.  As  a  result,  the  list  of  DoD  offices  and  DoD  contractors  who  use  E&V  technology  is  ever 
growing.  However,  steps  must  be  taken  to  ensure  the  continued  successful  application  of  E&V  technolo¬ 
gy  and  to  fully  exploit  the  effort  already  expended.  Specifically,  the  DoD  must: 

•  Transition  existing  DoD-developed  E&V  technology  to  an  agency  where  it  can  be 
maintained,  matured,  and  made  available  to  users. 

•  Promote  the  development  of  additional  E&V  technology  (The  Tools  and  Aids 
Document  prioritzes  and  details  the  required  technology.). 

•  Encourage  the  use  of  E&V  technology  through  acquisition  policy  changes  and  die 
development  of  Government  “centers  of  expertise.” 

Currently,  the  E&V  Project  is  managed  by  the  U.S.  Air  Force  Avionics  Laboratory.  Although  the 
laboratory  is  an  excellent  choice  for  technology  development,  its  charter  is  not  conducive  to  long  term 
maintenance  of  technical  products .  Additionally,  the  laboratory  infrastructure  is  not  set  up  to  support  the 
widespread  distribution  requirements  necessary  for  existing  and  future  E&V  technology.  Therefore,  a 
DoD  agency  or  agencies  must  be  identified  and  agree  to  accept  the  current  E&V  Project  developments 
for  the  purpose  of  maintenance,  enhancement,  and  distribution. 

It  is  my  belief  that  the  E&V  Project  has  barely  scratched  the  surface  in  developing  the  E&V  tech¬ 
nology  required  by  the  DoD.  With  the  availability  of  a  document  such  as  die  Tools  and  Aids  Document,  the 
only  other  items  required  for  developing  additional  E&V  technology  are  experienced  personnel  and 
money.  Therefore,  it  is  recommended  that  the  DoD  identify  an  agency  or  agencies  to  continue  the  develop¬ 
ment  ofEA  V  technology  and get  that  agency  to  commit  to  the  process.  To  exploit  the  DoD ’s  current  experi¬ 
ence  base,  it  is  highly  recommended  that  this  action  take  place  prior  to  the  termination  of  the  E&V  Project. 

In  today ’s  shrinking  DoD  budget  climate,  we  cannot  afford  the  software  development  failures  of 
the  past.  Many  of  these  failures  can  be  directly  attributed  to  the  acquisition  and  attempted  use  of  func¬ 
tionally  deficient  software  tools.  The  application  of  current  E&V  technology  can  alleviate  many  of  these 
problems  with  respect  to  compilation  systems,  and  to  a  lesser  degree  with  other  software  tools.  However, 
the  application  of  E&V  technology  is  not  without  its  own  problems. 

First,  most  software  tools  acquisition  personnel  are  not  aware  of  the  available  technology.  If 
they  are  aware ,  they  are  not  sure  how  to  leverage  it  during  the  acquisition  process .  This  is  a  publicity  and 
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education  problem.  Second,  application  of  E&V  technology,  such  as  the  AC  EC,  is  recommended  for  the 
technically  proficient.  There  are  learning  curves  associated  with  understanding  how  to  use  the  technolo¬ 
gy  and  interpret  its  results.  Finally,  today  there  are  a  significant  number  of  individual  organizations 
attempting  to  use  E&V  technology.  There  is  no  current  evidence  that  the  organizations  performing 
software  tool  evaluations  are  sharing  their  experiences  and  results.  Thus,  different  agencies  could  be 
evaluating  the  very  same  products. 

For  these  reasons,  it  is  recommended  that  the  DoD  identify  an  agency  or  agencies  to  (1)  be 
responsible  for  the  application  of  E&V  technology,  and  (2)  acquire  or  develop  the  expertise  necessary  to 
do  so.  This  agency  could  be  the  same  one  selected  to  maintain  and  enhance  existing  E&V  technology. 
This  organization  would  also  be  responsible  for  proliferating  E&V  technology  information  throughout 
the  software  tool  acquisition  community  and  providing  the  community  with  the  guidance  necessary  to 
exploit  E&V  technology  during  the  acquisition  process. 
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3.  WORKING  GROUP  REPORTS 

This  section  contains  a  brief  history  of  each  working  group  chartered  within  the  E&V  Team. 
Included  in  the  reports  is  the  purpose  of  each  group,  products  delivered  by  each  group,  a  general  discus¬ 
sion  of  each  group’s  activities,  and  specific  recommendations  made  by  each  working  group.  These 
recommendations  supplement  those  made  by  the  E&V  Team  as  a  whole  and  presented  in  Section  4. 

3.1  ADA  COMPILER  EVALUATION  CAPABILITY  WORKING  GROUP  (ACECWG) 

3.1.1  ACECWG  Purpose 

The  purpose  of  the  ACECWG  was  to  provide  feedback  to  the  ACEC  contract  monitor  pertaining 
to  ( 1 )  two  versions  of  the  ACEC  software  product  and  its  documentation,  and  (2)  plans  for  maintenance , 
enhancement,  and  use  of  the  ACEC. 

3.1.2  ACECWG  Mayor  Products 

While  tire  ACECWG  did  not  produce  a  formal  product,  the  ACEC  was  influenced  by  die 
comments  and  recommendations  of  the  ACECWG.  Tire  ACEC  is  described  in  Section  2.4.1. 

3.1  J  ACECWG  Activities 

The  ACECWG  had  an  important  and  beneficial  impact  on  the  ACEC  software  product  and  its 
documentation.  The  group  played  a  central  role  in  defining  the  scope  of  the  Ada  compiler  evaluation 
problem  and  a  taxonomy  of  evaluation  issues  against  which  the  coverage  of  the  ACEC  could  be 
measured.  Major  weaknesses  in  the  coverage  achieved  by  the  ACEC  were  pointed  out  and  drafts  of  die 
documentation  were  critiqued.  When  Version  1.0  of  the  software  product  was  released,  ACECWG 
members  ran  die  tests  and  provided  software  trouble  reports  and  justification  for  modifications.  It  is  safe 
to  say  that  the  ACECWG  provided  the  earliest  and  the  most  in-depth  commentary  on  the  test  suite,  its 
documentation,  and  its  suitability  for  use.  In  particular,  the  ACECW G  prioritized  the  ACEC  1 .0  problem 
reports  according  to  their  severity  and  provided  advice  on  the  level  of  quality  required  for  the  ACEC 
before  it  should  be  considered  suitable  as  a  component  for  a  Quality  Testing  Service. 

While  the  group  had  no  input  into  the  original  statement  of  work  for  ACEC  Version  1 .0,  it  did  sig¬ 
nificantly  influence  the  contents  of  Version  2.0.  Additions  to  Version  2.0  included  300  new  performance 
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tests  as  well  as  assessment  capabilities  for  three  new  functional  areas — diagnostics,  the  debugger,  and 
the  program  library  system.  The  group  contributed  even  more  to  suggested  enhancements  and  improve¬ 
ments  to  ACEC  that  will  influence  Version  3.0.  In  fact,  there  will  be  major  revisions  in  Version  3.0  that 
will  include  adding  compile  time,  capacity,  and  memory  size  tests,  as  well  as  addressing  some  of  the  crit¬ 
ical  usability  issues  associated  with  test  organization  and  naming  conventions,  and  simplification  of  the 
operating  procedures.  These  usability  issues  must  be  addressed  in  order  to  give  the  maximum  amount  of 
information  while  reducing  the  amount  of  time  required  to  learn  and  run  the  evaluation  suite. 

The  ACECWG  also  provided  considerable  feedback  on  several  iterations  of  a  plan  for  an  evalua¬ 
tion  service  based  on  the  ACEC.  This  service  has  most  recently  been  referred  to  as  the  Quality  Testing 
Service.  As  a  result  of  ACECWG  and  general  E&V  Team  input,  the  plan  was  modified  substantially. 
Originally  the  draft  policy  was  modelled  after  the  Ada  Compiler  Validation  Capability  (ACVC)  and  the 
Ada  Validation  Service  and  was  called  the  Ada  Compiler  Evaluation  Procedures  and  Guidelines  (PAG) 
Version  1 .0,  dated  01  Mar  89.  After  much  input  from  die  E&V  Team  and  others  this  became  the  Ada 
Compiler  Performance  Testing  Service  (PTS)  Procedures,  dated  15  Aug  89.  After  more  extensive  re¬ 
view  and  comment  this  became  the  Quality  Testing  Service  (QTS)  Procedures,  dated  06  Oct  89.  While 
not  yet  perfect,  the  latest  version  represents  a  very  substantial  improvement  over  the  original  effort  and, 
in  the  opinion  of  the  ACECWG,  could  serve  as  the  basis  for  a  DoD  Ada  compiler  evaluation  process. 

3.1.4  ACECWG  Recommendations 

None  beyond  those  presented  in  Section  4. 


3.2  CAIS  IMPLEMENTATION  VALIDATION  CAPABILITY  WORKING  GROUP 
(CTVCWG) 

3.2JL  CIVCWG  Purpose 

The  CIVCWG  was  responsible  for  examining  and  supporting  validation  of  the  CTVC  products. 
Thus,  the  CIVCWG  provided  a  forum  for  technical  review  and  comment  on  the  CAIS  Implementation 
Validation  Capability  (CIVC)  and  CIVC2  contractual  products  for  CAIS  and  for  C  AIS-A,  respectively. 
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3.2.2  CIVCWG  Major  Products 

While  the  CIVCWG  did  not  produce  a  formal  product,  the  CIVC,  Versions  1  and  2,  were 
influenced  by  the  comments  and  recommendations  of  the  CIVCWG.  The  CIVC  is  described  in 
Section  2.4.2. 

3.23  CIVCWG  Activities 

The  CTVCWG ’s  primary  activity  was  that  of  providing  technical  input  to  the  CTVC  contractor  to 
improve  the  quality  and  usability  of  the  CIVC  product.  The  comments,  critiques,  and  recommendations 
were  provided  via  regularly  scheduled  quarterly  E&V  Team  meetings,  contract-required  Technical 
Interchange  Meetings  (TIM),  and  extensive  use  of  the  electronic  mail  facility  on  the  MILNET.  Several 
of  the  CIVCWG  members  participated  in  the  standardization  activities  and  reviews  for  both  CAIS 
(DoD-STD- 1938)  and  CAIS- A  (MIL-STD-1838A)  and  are  currently  active  in  the  standardization  activ¬ 
ity  for  the  Portable  Common  Interface  Set  (PCIS).  Consequently,  the  necessary  expertise  has  been  avail¬ 
able  to  ensure  the  development  of  a  usable  and  relevant  validation  capability  for  CAIS  and  CAIS-A. 
Additionally,  the  CIVCWG  reviewed  and  commented  on  the  methods  and  techniques  being  developed 
to  achieve  die  validation  capability.  During  early  phases  of  the  team’s  activity,  this  working  group 
operated  under  the  name  of  the  CAIS  Validation  Capability  Working  Group  (CVCWG). 

3.2.4  CIVCWG  Recommendations 

3.2.4.1  The  E&V  technology  development  and  maintenance  efforts  related  to  standards  should 
be  transferred  to  a  Government  funded  organization,  such  as  a  Federally  Funded  Research  Center 
(FFRC). 

The  CTVC2  contract  activity  continues  until  March  1992  and  will  continue  to  require  technical 
review  as  well  as  continued  funding.  The  full  development  of  this  validation  capability  will  enhance  the 
usability  of  implementations  of  the  CAIS-A  standard  as  well  as  the  confidence  of  die  users  of  the 
standard.  Politically  and  economically,  it  may  be  more  achievable  to  arrange  for  an  existing  FFRC  (e.g., 
the  SEI)  to  acquire  the  charter  for  E&V  technology.  Future  enhancements  of  E&V  technology  should 
concentrate  on  supporting  large  software  systems  development,  particularly  with  respect  to  the 
following  issues:  evaluating  the  interactions  of  multiple  APSEs,  evaluating  and  validating  changing 
configurations  of  evolving  APSEs,  and  evaluating  and  validating  the  integration  of  tools  developed  for 
use  on,  or  tailored  to,  a  particular  project. 
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3-3  CLASSIFICATION  WORKING  GROUP  (CLASSWG) 

33.1  CLASSWG  Purpose 

The  purpose  of  the  CLASSWG  was  to  provide  feedback  to  the  E&V  Technical  Support  contract 
monitor  on  the  E&V  Reference  System.  This  included  feedback  on  the  draft  versions  of  the  Reference 
Manual  and  Guidebook,  as  well  as  feedback  on  the  plans  for  maintenance,  enhancement,  and  use  of  the 
Reference  System.  Specifically,  the  CLASSWG  was  to  serve  as  the  focal  point  for  analysis  of  the  E&V 
Reference  System,  solicit  information  and  recommendations  regarding  E&V  technology,  classify  E&V 
technology  for  inclusion  in  the  Reference  System,  aid  in  technology  transition  of  the  Reference  System, 
delineate  issues  associated  with  evaluating  APSEs  when  taken  as  a  whole,  and  recommend  new  areas  of 
investigation. 

333  CLASSWG  Major  Products 

While  die  CLASSWG  did  not  produce  a  formal  product,  the  E&V  Reference  System  was 
influenced  by  the  comments  and  recommendations  of  the  CLASSWG.  The  E&V  Reference  System  is 
described  in  Section  2.4.3.  In  addition  to  providing  suggestions  for  ways  to  improve  die  E&V  Reference 
System ,  the  CLASSWG  participated  directly  in  drafting  new  sections  of  the  documents,  including  some  of 
the  checklists  and  questionnaires  which  comprise  the  assessment  technology  found  within  the 
Guidebook. 

3 33  CLASSWG  Activities 

All  of  the  activities  of  the  CLASSWG  were  devoted  to  making  the  E&V  Reference  System  meet 
the  needs  of  DoD  and  its  contractors.  Specifically,  the  CLASSWG  reviewed  drafts  of  the  Reference  Sys¬ 
tem,  identified  areas  where  additional  technology  was  needed,  created  new  technology  in  the  form  of 
questionnaires  and  checklists,  solicited  the  help  of  the  E&V  Team  members  with  expertise  in  areas  other 
than  those  of  die  CLASSWG,  reviewed  external  sources  of  information  to  identify  new  technology  to  be 
summarized  in  the  Reference  System,  created  new  entries  and  synopses,  and  continuously  searched  for 
new  E&V  technology  during  daily  activities  in  their  home  organizations  and  elsewhere.  During  the  early 
years  of  the  team ’s  activity  an  APSE  Working  Group  (APSEWG)  produced  materials  that  influenced  the 
classification  schema  which  now  provides  a  structure  for  the  E&V  Reference  Manual. 

3-3.4  CLASSWG  Recommendations 

33A1  Establish  a  review  group  of  professionals  with  diverse  backgrounds  to  support  the 
continued  enhancement  of  the  E&V  Reference  System. 
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An  effort  such  as  that  required  to  produce  and  enhance  what  is  intended  to  be  a  comprehensive  Ref¬ 
erence  System  for  E&V  technology  requires  familiarity  with  advances  and  expertise  in  a  wide  range  of 
topics.  To  date,  the  E&V  Reference  System  has  benefitted  from  the  inputs  and  reviews  of  the  CLASS  WG 
and  the  E&V  Team,  as  a  whole.  It  will  be  very  difficult  to  compensate  for  the  loss  of  these  resources  with 
supplemental  input  from  within  a  single  corporation.  Therefore,  it  is  recommended  that  an  external  re¬ 
view  group  be  established  to  support  the  continued  enhancement  of  the  E&V  Reference  System. 


3.4  REQUIREMENTS  WORKING  GROUP  (REQWG) 

3.4.1  REQWG  Purpose 

The  purpose  of  die  REQWG  was  to  identity  requirements  for  the  activities  of  the  E&V  Team  and 
for  APSE  evaluation  and  validation.  Specifically,  this  included  maintaining  the  E&V  Requirements 
Document;  analyzing  the  requirements  to  determine  adequacy,  completeness,  traceability,  consistency, 
and  feasibility;  identifying  issues  which  may  impact  the  development  of  E&V  technology;  providing 
recommendations  for  the  acquisition  of  E&V  Tools  and  Aids;  and  preparing  position  papers  addressing 
issues  concerning  E&V  requirements. 

3.4.2  REQWG  Major  Products 

The  major  products  of  the  REQWG  included  the  Requirements  Document  [E&V  Requirements 
1987]  and  the  Tools  and  Aids  Document  [E&V  Tools  and  Aids  1990].  The  Requirements  Document  is 
described  in  Section  2.3.1.  The  Tools  and  Aids  Document  is  described  in  Section  2.3.5. 

3.4 3  REQWG  Activities 

REQWG  activities  included  defining  requirements  for  the  E&V  Team  and  the  E&V  Project, 
periodically  reviewing  E&V  efforts  to  determine  whether  or  not  requirements  were  being  satisfied,  de¬ 
fining  future  needs  for  E&V  technology,  periodically  determining  which  team  members  were  attending 
conferences  so  that  E&V  materials  could  be  distributed,  creating  die  E&Ving  Newsletter  -  a  newsletter 
about  E&V  technology  updates,  and  identifying  opportunities  for  E&V  technology  transition. 

3.4.4  REQWG  Recommendations 

None  beyond  those  presented  in  Section  4. 
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3.5  STANDARDS  EVALUATION  AND  VALIDATION  WORKING  GROUP  (SEVWG) 

3.5.1  SEVWG  Purpose 

The  purpose  of  the  SEVWG  was  to  provide  a  forum  for  discussions  pertaining  to  the  evaluation 
and  validation  of  current,  proposed,  and  future  APSE  related  standards  and  their  implementations. 
Included  in  the  charter  were  the  requirements  to:  ( 1 )  identify  issues  relating  to  validating  conformance  to 
an  APSE  related  standard,  and  (2)  suggest  approaches  for  achieving  conformance.  The  SEVWG  was 
also  concerned  with  all  aspects  of  evaluating  implementations  of  standards.  Both  technical  and 
non-technical  aspects  of  APSE  standards  were  considered. 

3.5.2  SEVWG  Major  Products 

The  major  products  of  the  SEVWG  include  the  APSE  Component  Validation  Procedures 
Document  [CVP 1985],  the  Issues  and  Strategies  for  CAIS  Evaluation  and  Validation  document,  [Issues 
1987]  and  the  Issues  and  Strategies  for  Evaluation  and  Validation  of  CAIS-A  Implementations  docu¬ 
ment  [Issues  1990].  The  APSE  Component  Validation  Procedures  Document  is  described  in 
Section  2.3.2.  The  Issues  and  Strategies  for  CAIS  Evaluation  ami  Validation  document  is  described  in 
Section  2.3.3.  The  Issues  and  Strategies  for  Evaluation  and  Validation  of  CAIS-A  Implementations 
document  is  described  in  Section  2.3.4. 

353  SEVWG  Activities 

The  SEVWG  began  as  the  Common  APSE  Interface  Set  Working  Group  (CAISWG),  with  a 
limited  initial  chatter  that  included  only  the  examination  of  issues  concerning  the  validation  of  the  CAIS 
interfaces.  The  E&V  Team’s  CAISWG  produced  die  APSE  Component  Validation  Procedures  Docu¬ 
ment  in  1985.  Realizing  that  several  environment  standards  would  evolve,  the  CAISWG  changed  its 
name  to  SEVWG  to  reflect  a  broader  role  in  looking  at  various  APSE  standards.  Although  the  SEVWG 
has  considered  and  discussed  other  APSE  standards,  its  products  have  focused  on  the  CAIS.  The 
CAIS-A  is  a  design  enhancement  of  DoD-STD- 1 838  and  has  been  standardized  as  MIL-STD-1838 A.  It 
was  developed  by  SofTech,  San  Diego  under  contract  to  the  Naval  Ocean  Systems  Center.  In  February 
1990,  the  SEVWG  released  its  Issues  and  Strategies  for  Evaluation  and  Validation  of  CAIS-A  Imple¬ 
mentations,  an  analysis  of  the  validateability  of  CAIS-A. 

351.4  SEVWG  Recommendations 

None  beyond  those  presented  in  Section  4. 
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4.  RECOMMENDATIONS 

After  seven  years  of  research,  development,  evaluation,  and  lively  debates,  the  E&V  Team  has 
settled  on  the  following  recommendations  as  being  the  most  critical  for  action.  The  list  of  recommenda¬ 
tions  is  summarized  in  Table  4- 1 .  Some  of  die  recommendations  are  meant  to  compensate  for  the  antici¬ 
pated  absence  of  the  team’s  involvement  in  future  E&V-related  activities.  Some  indicate  actions  re¬ 
quired  for  the  successful  application  of  E&V  technology  on  DoD  programs.  Many  are  just  now  becoming 
appropriate  due  to  the  development  of  technology  by  the  E&V  Project.  Still  others  reflea  areas  in  need  of 
future  enhancement.  It  is  the  sincere  desire  of  the  E&V  Team  that  the  AJPO  will  carefully  consider  each  of 
these  recommendations,  identify  appropriate  means  of  implementation,  end  work  with  related  organiza¬ 
tions  to  advance  and  enhance  the  technology  and  its  use  to  the  benefit  of  the  DoD.  Our  commitment  to  the 
technology  should  be  evident  given  our  ability  to  convince  our  home  organizations  that  the  voluntary 
support  of  this  effort  should  be  continued  over  a  seven  year  period.  It  is  hoped  that  the  AJPO ’s  actions  will 
leverage  our  contributions  and  indicate  an  equal  or  greater  commitment  to  die  technology  in  the  future. 


4.1  RAISE  PUBLIC  AWARENESS  OF  E&V  AND,  AS  A  BROAD  OBJECTIVE, 

ENCOURAGE  USE  OF  APSE  E&V  TECHNOLOGY 

•  Support  software  process  insertion  activities 

•  Stress  utilization  of  existing  technology 

•  Support  educational  activities 

•  Provide  incentives 

•  Incorporate  into  existing  policies  and  standards. 

Efforts  to  insert  E&V  technology  into  mainstream  software  development  processes  should  be 
supported.  Insertion  of  E&V  technology  requires:  ( 1 )  Availability  of  the  technology,  (2)  Availability  of  a 
skill  base  with  respea  to  the  application  of  the  technology  and  (3)  Perceived  Value  in  the  application  of 
the  technology.  As  discussed  below,  each  of  these  items  is  within  reach.  Therefore,  it  is  recommended 
that  specific  major  programs  (e.g.,  STARS,  SDI,  ATF)  be  targeted  for  the  first  insertion  of  assessment 
technology  in  the  procurement  process.  As  a  secondary  effort,  it  is  recommended  that  major  on-going 
programs  develop  a  mechanism  by  which  assessment  technology  is  applied  to  determine  the  current 
quality  of  those  programs  and  to  identify  areas  for  improvement. 
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Table  4-1  Summary  of  E&V  Team  Recommendations 

1.  Raise  Public  Awareness  of  E&V  and,  as  a  Broad  Objective,  Encourage  Use  of 
APSE  E&V  Technology 

•  Support  software  process  insertion  activities 

•  Stress  utilization  of  existing  technology 

•  Support  educational  activities 

•  Provide  incentives 

•  Incorporate  into  existing  policies  and  standards. 

2.  Mandate  Compiler  Evaluations 

•  Integrate  compiler  evaluation  with  procurements,  not  with  the  validation  process 

•  Require  appropriate  evaluations,  considering  project  size,  nature,  and 
critical  factors. 

3.  Create  a  Quality  Testing  Center  of  Expertise 

•  Perform  evaluations  upon  request  and  provide  evaluation  resources  such  as 
software,  documentation,  and  guidelines. 

•  Provide  consulting  services  to  projects.  Government  agencies,  and  contractors. 

4.  Continue  Development  of  E&V  Technology  in  Four  Areas  and  Coordination  with 
Related  Activities 

•  Enhance  the  APSE  E&V  Reference  System  and  develop  a  hypertext  version. 

•  Enhance  the  Ada  Compiler  Evaluation  Capability  (ACEC). 

•  Enhance  the  CAIS  Implementation  Validation  Capability  (dVQ  and 
transition  to  PCIS  validation 

•  Continue  development  of  integrated  APSE  evaluation  capability  (structured 
experiments). 

3.  Promote  Applicable  APSE  Standards 

•  Promote  standards  for  tool  transportability,  data  interoperability,  data 
representation,  user  interfaces,  communications,  Ada  runtimes,  etc. 

•  Stress  importance  of  such  standards  as  a  benefit  to  software  maintenance 
organizations. 

6.  Identify  an  Appropriate  Organization  for  APSE  E&V  Standards  Activities;  Develop, 
Manage,  and  Maintain  E&V  Test  Suites  Using  A  Unified  Approach 

•  Develop  a  single  approach  and  encourage  use  of  tools  in  developing  suites. 

•  Require  test  suite  developers  to  deliver  suites  in  a  maintainable  form,  such  that 
appropriate  tools  can  be  used. 

7.  Promote  International  Sharing  of  E&V  Technology,  But  Avoid  Joint  Multinational 
Development  of  Assessor  Products 
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Today,  a  wide  variety  erf  APSE  E&V  technology  is  available  for  use.  Some  of  the  existing  capa¬ 
bilities  were  developed  by  the  E&V  Project  (e.g.,  the  ACEC);  some  were  developed  independent  of  this 
effort  (e.g.,  the  PIWG  benchmarks).  While  the  E&V  Reference  System  is  far  from  complete,  it  does 
contain  and  reference  valuable  E&V  technology  in  many  important  areas.  These  include  compilation 
system  assessors,  requirements  and  design  tool  assessors,  and  assessors  that  are  applicable  to  all  tools 
(e.g.,  cost,  required  configuration,  and  licensing  issues  questionnaires).  Widespread  use  of  these  tech¬ 
niques  will  assist  the  Government  and  its  contractors  in  selecting  suitable  tools  and  will  improve  the 
quality  of  tools  produced  by  the  vendors.  This  is  particularly  important  in  an  environment  in  which 
increasing  emphasis  is  being  placed  on  buying  more  software  off  the  shelf  rather  than  building  it  from 
scratch.  The  E&V  Reference  System  should  be  reviewed  to  determine  what  capabilities  exist  before 
efforts  to  develop  new  technology  are  initiated.  Future  developments  should  build  upon  material  already 
reported  in  the  Reference  System,  rather  than  replicate  it. 

The  availability  of  software  engineers  with  the  skills  needed  to  apply  E&V  technology  and  inter¬ 
pret  the  results  can  be  improved  by  supporting  an  aggressive,  formal  effort  to  educate  the  technical 
public  about  the  nature  and  availability  of  E&V  technology.  When  providing  E&V  education,  however, 
several  key  aspects  must  be  considered.  Defining  die  terminology  of  validation  and  evaluation  by 
describing  what  they  are,  what  they  are  not,  how  they  differ,  and  how  they  relate  to  one  another  must  be 
foremost.  Further,  the  benefits  of  using  the  technology  should  be  clearly  discussed.  Available  E&V 
technology  should  also  be  presented  and  demonstrated.  There  are  several  avenues  in  existence  today 
which  should  be  used  to  support  such  E&V  education.  A  significant  resource  is  the  Software  Engineer¬ 
ing  Institute  (SEI)  which  has  already  developed  a  software  engineering  curriculum.  In  addition,  die 
ASEET,  CRAC,  and  special  educational  briefings  by  the  Services  should  be  used  as  forums  for  improv¬ 
ing  public  awareness  of  E&V  technology.  Special  emphasis  should  be  placed  on  educating  DoD 
procurement  personnel,  as  to  what  assessment  technology  is,  what  assessment  tools  exist,  and  how 
assessment  technology  can  be  applied  to  varying  aspects  of  proposal  or  performance  evaluations. 
Successful  E&V  education  depends  on  the  development  and  maintenance  of  appropriate  presentation 
materials.  To  this  end,  a  curriculum  module  on  E&V  technology  should  be  developed  and  updated  peri¬ 
odically.  Such  a  module  would  provide  a  single  formal  package  of  materials  which  could  be  used 
throughout  the  industry  for  E&V  education  and  information. 

Perceived  value  can  be  increased  by  customers  illustrating  their  commitment  and  enthusiasm  via 
the  employment  of  incentives  for  the  use  of  E&V  technology.  The  Ada  programming  language  was 
designed,  in  part,  to  foster  good  software  engineering  practices .  The  use  of  APSE  E&V  technology  should 
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be  one  of  those  “good”  practices.  It  is  believed  that  the  application  of  E&V  (assessment)  technology  will 
provide  a  dramatic  reduction  in  acquisition/maintenance  costs.  Therefore,  contract  incentive  fees  should 
be  employed  to  ensure  that  E&V  receives  the  management  attention  required  to  accomplish  the  integra¬ 
tion  of  the  technology  into  existing  processes.  Another  type  of  incentive  would  encourage  the  application 
of  assessment  technology  prior  to  contract  award.  When  requests  for  proposal  are  created  by  the  DoD, 
they  should  specify  the  assessment  technology  data  that  must  be  submitted  with  the  contractor ’s  proposal 
for  use  as  technical  evaluation  criteria  and/or  contract  award  factors.  This  assessment  data  would  then  be 
used  as  feasibility  or  “goodness”  indicators  when  the  proposal  is  being  technically  evaluated. 

Finally,  a  high-level  policy  encouraging  the  use  and  continued  development  of  E&V  products 
should  be  implemented  throughout  the  DoD.  Upper  echelon  support  is  critical  for  continued  emphasis 
within  the  Services  and  other  DoD  agencies.  Strong  guidance,  coupled  with  other  incentives,  is  neces¬ 
sary  to  incorporate  E&V  technology  into  common  practice.  Existing  standards  and  directives  should 
include  instructions  for  E&V  technology  use.  For  example,  the  Software  Development  Plan  required  by 
DoD-STD-2167A  should  be  modified  to  require  developers  to  describe  their  planned  use  of  E&V  tech¬ 
nology.  Industry  must  be  encouraged  to  incorporate  this  technology  into  the  early  phases  of  their 
software  development  processes.  Tire  exploitation  of  existing  policies  and  standards  can  be  an  avenue 
by  which  to  accomplish  this  goal. 


4.2  MANDATE  COMPILER  EVALUATIONS 

•  Integrate  compiler  evaluation  with  procurements,  not  with  the  validation  process 

•  Require  appropriate  evaluations,  considering  project  size,  nature,  and  critical 
factors. 

When  the  DoD  developed  the  Ada  language,  it  also  developed  a  policy  for  the  validation  of 
language  implementations  against  the  standard.  However,  we  have  all  come  to  realize  that  the  validation 
of  a  compiler  does  not  necessarily  mean  that  it  is  fit  for  use  on  a  specific  project.  Large  Ada  software 
development  projects,  if  well  managed,  would  perform  compiler  evaluations  at  different  points  in  the 
life  cycle  for  different  reasons.  For  example,  early  evaluations  would  support  the  compiler/vendor  selec¬ 
tion  process.  Later  evaluations  are  appropriate  when  the  actual  target  configuration  characteristics  and 
settings  have  changed;  they  serve  to  keep  the  development  team  (and  maintenance  team)  fully  aware  of 
the  current  system ’s  strengths  and  weaknesses.  Such  evaluations  are  likely  to  make  use  of  the  ACEC  test 
suite  and  other  benchmark  tests  tailored  for  the  specific  application,  as  well  as  additional  qualitative 
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assessments  based  on  checklists  and  questionnaires  available  via  the  E&V  Reference  System.  The  de¬ 
tails  of  such  evaluations  will  vary,  however,  depending  on  each  project’s  critical  success  factors,  size, 
application  domain,  etc. 

A  policy  should  be  established  requiring  that,  for  each  procurement,  the  critical  compilation 
system-related  issues  be  determined  and  candidate  compilers  evaluated.  Program  personnel  should  be 
required  to  identify  the  risk  drivers  that  the  compiler  presents  to  the  program  and  structure  the  evaluation 
process  to  minimize  the  risk.  Program  offices  should  also  be  required  to  document  the  approach  used  to 
select  a  compiler  to  minimize  procurement  risk. 


4  J  CREATE  A  QUALITY  TESTING  CENTER  OF  EXPERTISE 

•  Perform  evaluations  upon  request  and  provide  evaluation  resources  such  as  soft¬ 
ware,  documentation,  and  guidelines. 

•  Provide  consulting  services  to  projects.  Government  agencies,  and  contractors. 

The  process  of  evaluating  APSEs  as  a  whole  or  individual  components  of  an  APSE  can  be  quite 
complex.  Each  APSE  component  provides  different  functionality  that  may  be  required  to  interact  with 
other  APSE  components  in  a  variety  of  ways.  This  may  be  the  subject  of  evaluation.  In  addition  to  die 
functionality  provided,  the  suitability  of  individual  tools  for  use  oa  a  major  development  may  be  of 
interest.  The  skills  required  to  use  available  APSE  E&V  technology  also  vary  based  on  the  subject  and 
process  of  the  evaluation.  In  some  cases,  it  may  be  sufficient  to  complete  a  checklist  describing  function¬ 
al  capabilities.  In  other  cases,  the  execution  of  a  test  suite  such  as  the  ACEC  and  die  detailed  analysis  of 
its  results  may  be  required  to  answer  the  questions  of  interest.  Since  procurements  of  APSE  components 
will  be  based  on  die  results  of  evaluations,  it  is  absolutely  crucial  that  the  evaluation  approach  employed 
is  appropriate,  and  the  analysis  is  correct  for  the  circumstances  involved. 

Due  to  the  variety  and  complexity  of  both  the  components  requiring  evaluation  and  the  technolo¬ 
gy  available  to  perform  the  evaluations,  it  is  recommended  that  a  Quality  Testing  Center  of  Expertise  be 
established.  It  should  be  a  resource  to  the  DoD  that  could  (1)  provide  evaluation  technology  to 
organizations  interested  in  performing  evaluations,  (2)  help  projects,  Government  agencies,  or 
contractors  determine  what  evaluations  are  necessary  and  what  technology  is  applicable,  (3)  evaluate 
the  process  and  results  of  evaluations  conducted  by  others  (for  example,  as  pan  of  a  procurement  activ¬ 
ity),  or  (4)  perform  evaluations  in  house,  on  request,  (providing  recommendations  to  its  customers 
concerning  APSE  technologies  appropriate  for  use  on  a  given  project). 
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4.4  CONTINUE  DEVELOPMENT  OF  E&V  TECHNOLOGY  IN  FOUR  AREAS  AND 

COORDINATION  WITH  RELATED  ACTIVITIES 

•  Enhance  the  APSE  E&V  Reference  System  and  develop  a  hypertext  version. 

•  Enhance  the  Ada  Compiler  Evaluation  Capability  (ACEC). 

•  Enhance  the  CAIS  Implementation  Validation  Capability  (CIVC)  and  transition  to 
PCIS  validation. 

•  Continue  development  of  integrated  APSE  evaluation  capability  (structured 
experiments). 

Existing  evaluation  and  validation  technology  is  still  incomplete  and  immature.  Contractual 
efforts  to  develop  and  refine  the  technology  should  be  continued.  Some  of  the  efforts  sponsored  by  the 
E&V  Project  have  been  through  several  iterations,  while  others  are  in  their  early  stages.  Still,  the  need 
for  E&V  technology  is  largely  unfulfilled.  As  a  result.  Government  programs  are  using  untried  and 
untested  tools  which  are,  in  many  instances,  responsible  for  costly  overruns  and  schedule  delays.  The 
E&V  technology  that  has  been  developed  to  date  is  helpful,  but  in  need  of  further  refinement  New  func¬ 
tionality  must  be  added  and  user  interfaces  improved.  For  a  relatively  small  incremental  investment  in 
existing  evaluation  and  validation  technology  for  APSE  tools,  considerable  expenditures  over  the  life 
cycles  of  all  software  programs  can  be  eliminated.  Among  the  products  still  under  development  by  die 
E&V  Project  are  the  E&V  Reference  System,  the  ACEC,  the  CIVC,  and  structured  experiments 
designed  to  evaluate  integrated  APSE  capabilities.  Failure  to  follow  up  on  these  efforts  and  carry  diem 
through  to  a  successful  conclusion  would  be  a  tremendous  waste  of  potentially  useful  technology. 

The  E&V  Reference  System  consists  of  two  coordinated  documents:  die  E&V  Reference 
Manual  [E&V  RM  1989]  and  the  E&V  Guidebook  [E&V  GB  1989].  These  documents  are  a  valuable 
source  of  information  that  describe  the  current  state  of  E&V  technology,  contain  some  of  die  technology 
(e.g.,  checklists  and  questionnaires)  within  their  covers,  provide  many  summaries  and  references  to 
detailed  descriptions  of  specific  instances  (e.g.,  test  suites)  of  the  technology,  provide  a  framework  for 
understanding  environments  and  their  assessment,  and  provide  definitions  of  die  terminology  used  in  all 
of  die  above.  The  Reference  System  has  been  updated  annually  since  1 987.  Since  E&V  technology  is  rap¬ 
idly  growing  and  improving,  it  is  necessary  to  continue  to  update  the  Reference  System  and  to  make  it 
readily  and  broadly  available.  In  particular,  future  evolution  of  the  E&V  Reference  System  should  be 
directed  at  issues  such  as  integrated  environments,  environment  infrastructures,  and  standards  validation 
and  evaluation.  There  is  no  “silver  bullet”  [Brooks  1987]  for  solving  the  software  crisis.  The  solution 
requires  more  than  a  single  tool  or  development  language  such  as  Ada.  Integration  or  cooperation  among 
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tools  in  an  environment  is,  however,  a  potentially  significant  contributor  to  future  improvements.  As  the 
emphasis  of  tool  and  environment  builders  shifts  toward  integration  issues,  the  emphasis  of  assessment 
technology  users  and  builders  should  shift  in  the  same  direction.  By  recognizing  the  shift  in  emphasis 
early,  the  E&V  Reference  System  can  help  accelerate  the  movement  toward  integrated  environments  by 
educating  the  user  community  on  the  important  E&V  issues.  The  usefulness  of  the  E&V  Reference 
System  can  be  further  enhanced  by  making  electronic  versions  of  the  system  widely  available  through 
repositories  such  as  the  STARS  repository  and  developing  an  on-line  version  with  hypertext  features. 

To  date,  two  versions  of  the  ACEC  have  been  released  to  the  public.  While  the  product  is  helpful 
in  its  current  form,  there  are  a  number  of  ways  to  make  it  more  useful.  The  first  version  of  the  ACEC 
stressed  performance  of  the  generated  code  and  the  Ada  runtime  system.  The  second  version  of  the 
ACEC  added  more  performance  tests;  provided  assessors  for  the  program  library  system,  diagnostics, 
and  the  debugger,  and  refined  its  analysis  tools.  Plans  exist  for  a  third  version  that  will  increase  the  atten¬ 
tion  paid  to  compile  time  performance,  capacity  testing,  and,  especially,  usability  issues.  Without  these 
improvements,  the  ACEC  will  be  considered  to  be  incomplete  and  hard  to  use.  Moreover,  die  evaluation 
suite  must  be  modified  to  address  die  changes  to  die  Language  Reference  Manual  [DoD  1983] 
anticipated  as  a  result  of  the  Ada9X  project.  The  ACEC  should  be  the  cornerstone  for  the  evaluation  of 
Ada  compilation  systems.  To  be  viable  in  this  role,  it  must  be  supported  over  the  long  term. 

The  CIV C  program  has  successfully  applied  new  technologies  to  product  development  in  support 
of  the  validation  testing  process.  Information  modeling  of  test  coverage  design  and  assessment  for  valida¬ 
tiontesting;  hypermedia  implementation  of  test  requirements,  test  designs,  and  their  traceability  relation¬ 
ships;  and  application  of  automated  methods  to  test  selection,  prioritization,  and  code  generation  have 
advanced  the  CTVC  validation  process  far  beyond  its  initial  capabilities.  These  validation  development 
tools  and  products  have  been  designed  to  accommodate  future  validation  needs  (i.e.,  PCIS),  and  addition¬ 
al  funding  is  crucial  in  order  to  continue  the  development  of  a  superior  validation  capability.  Considering 
that  (1)  the  automated  development  tools  are  prototype  systems  and  (2)  new  hypertext  systems  with  great¬ 
er  functionality  are  being  released  at  an  explosive  rate,  further  development  efforts  have  been  identified 
to  evolve  the  existing  CNC  into  a  state-of-the-art,  production  quality  validation  system. 

Most  of  the  E&V  technology  developed  to  date  is  oriented  toward  the  assessment  of  individual 
tools,  with  emphasis  on  compiler  validation  and  evaluation  (e.g.,  ACVC,  ACEC,  AES).  Now  that  Ada 
compilers  are  maturing,  attention  will  increasingly  turn  to  other  needs,  such  as  the  evaluation  of 
front-end  CASE  tools,  test  tools,  and  whole  APSEs.  An  especially  important  need  is  the  capability  to 
evaluate  APSEs  considered  as  integrated  systems  that  influence  the  productivity  of  whole  teams  working 
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throughout  the  entire  project  life  cycle.  A  promising  start  has  been  made  in  creating  a  family  of 
structured  experiments  that  are  designed  to  provide  this  kind  of  capability.  This  effort  should  be  contin¬ 
ued  and  expanded. 

The  coordination  ofE  &  V  technology  developments  with  related  activities  should  be  continued 
and  expanded.  There  are  at  least  three  kinds  of  Government-sponsored  activities  that  can  be  enhanced  by 
strong  coordination  with  E&V  activities.  First,  R&D  efforts  aimed  at  developing  better  and  more 
integrated  tools,  environments,  and  repositories  (e.g.,  STARS)  can  benefit  from  careful  consideration  of 
the  techniques  employed  to  evaluate  the  perfonnance  and  quality  of  these  elements.  Second,  major 
programs  which  must  evaluate  and  select  environment  components  (e.g.,  the  SDI,  the  NASA  Space 
Station)  can  benefit  from,  and  contribute  to,  the  latest  advances  in  evaluation  technology.  Their  environ¬ 
ment  selections  will  represent  major  investments  with  far-reaching  consequences.  Third,  activities 
related  to  the  definition  of,  or  mandated  use  of,  standards  (e.g.,  CAIS-A)  can  benefit  from  careful  consid¬ 
eration  of  validation  technology  and  die  problem  of  determining  conformance  to  a  standard.  To  date,  die 
E&V  Project  has  devoted  a  significant  amount  of  attention  to  coordination  with  related  efforts.  Numer  ¬ 
ous  briefings  and  birds-of-a-feather  sessions  have  been  held  at  Ada-related  conferences.  The  existence 
of  the  E&V  Team  was,  in  itself,  another  major  commitment  to  coordination.  Members  of  the  team  acted 
as  ambassadors  for  E&V  technology  in  their  home  organizations  and  in  all  of  die  related  activities  in 
which  they  were  involved.  In  addition,  they  represented  the  voice  of  reason  to  the  E&V  technology  de¬ 
velopment  efforts,  ensuring  that  the  end  products  would  be  appropriate  fur  its  intended  user  community. 
It  will  be  very  difficult  to  compensate  for  the  loss  of  30  ambassadors.  As  a  minimum,  managers  of  R&D 
efforts,  major  programs,  and  standards  activities  should  be  required  to  report  regularly  on  their  use  and 
knowledge  of  E&V  technology.  Similarly,  managers  of  E&V  technology  developments  should  be 
required  to  work  direcdy  with  selected  major  programs  to  ensure  realism  and  appropriateness  of 
ongoing  work. 

4J  PROMOTE  APPLICABLE  APSE  STANDARDS 

•  Promote  standards  for  tool  transportability,  data  interoperability,  data  representa¬ 
tion,  user  interfaces,  communications,  Ada  runtimes,  etc. 

•  Stress  importance  of  such  standards  as  a  benefit  to  software  maintenance  organiza¬ 
tions. 

Important  aspects  of  tool  reusability  and  maintainability  can  only  be  achieved  through  interface 
standardization.  Standards  should  be  developed  in  three  areas:  tool  interfaces,  ran  time  environment 
(RTE)  interfaces,  and  data  interchange  protocols. 
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Tool  transportability  interfaces  such  as  CAIS-A  and  PC  IS  are  currently  under  development  or  are 
in  use.  Such  standards  will  eventually  result  in  lower  maintenance  costs  of  Government-owned 
applications.  The  increased  portability  of  tools  will  reduce  personnel  training  costs  associated  with  tool 
usage  and  a  wider  potential  market  will  motivate  vendors  to  develop  better  and  more  effective  tools. 

Ada  RTEs  vary  significantly  in  capability  and  interfaces.  Standardization  will  facilitate  portabil¬ 
ity  of  applications  that  require  access  to  the  low  level  features  of  an  Ada  implementation.  In  particular, 
difficult  categories  of  applications  such  as  “hard  real  time”  applications,  will  receive  significant  benefits 
from  RTE  standardization.  The  activities  and  reports  generated  by  the  Ada  Run  Time  Environments 
Working  Group  (ARTEWG)  within  SIGAda  provide  a  valuable  foundation  for  standards  development 
in  this  area. 

Data  interchange  protocols  provide  the  capability  to  design  new,  highly  specialized,  tools  to 
interface  with  existing  toolsets  and  existing  maintenance  databases.  These  protocols  are  essential  to  long 
term  maintenance  activities  because,  with  well  documented  interchange  protocols,  new  tools  can  be 
created  to  support  systems  whose  developers  are  no  longer  in  the  marketplace.  Additionally,  as  vendors 
leave  the  market  or  reduce  support  on  older  toolsets  (APSEs)  new  vendors  can  target  the  same  market 
niche  and  customer  base  with  new  tools.  They  can  provide  equivalent  or  better  functionality  because  the 
data  protocols  will  be  well  documented  and  standardized. 

This  area  of  activity  is  not  yet  well  supported  in  the  Ada  community.  Consequently,  it  would  be 
appropriate  to  encourage  an  FFRC,  such  as  the  SEI,  to  initiate  research  in  this  area  and  to  act  as  an 
advocacy  group  for  the  standardization  of  data  interchange  protocols. 


4.6  IDENTIFY  AN  APPROPRIATE  ORGANIZATION  FOR  E&V  OF  APSE 

STANDARDS  ACTIVITIES;  DEVELOP,  MANAGE,  AND  MAINTAIN  E&V  TEST 
SUITES  USING  A  UNIFIED  APPROACH 


•  Develop  a  single  approach  and  encourage  use  of  tools  in  developing  suites. 

•  Require  test  suite  developers  to  deliver  suites  in  a  maintainable  form,  such  that 
appropriate  tools  can  be  used. 

APSE  standards  should  be  viewed  as  inseparable  from  the  mechanisms  required  to  assess 
conformance  and  quality  of  implementations,  and  appropriateness  for  use.  The  AJPO  should  continue 
efforts  to  provide  evaluation  and  validation  mechanisms  for  existing  Ada  and  APSE  standards .  Tire  early 
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introduction  of  E&V  technology  into  new  standards  efforts  will  have  the  following  effects:  increase  the 
confidence  of  users  in  the  new  standards,  avoid  design  decisions  that  inhibit  evaluation  and  validation, 
and  reduce  the  time  required  to  achieve  usable  standards  by  developing  assessment  technology  in  paral¬ 
lel  with  the  standards. 

An  organization  should  be  formed  for  developing,  maintaining,  and  promulgating  E&V  technol¬ 
ogy  for  APSE  standards.  The  organization  should  develop  a  single  approach  for  test  suite  development, 
management,  maintenance,  and  use,  which  accommodates  all  aspects  of  evaluation  and  validation  suites. 

A  number  of  future  standards  would  benefit  from  the  technology  developed  as  part  of  the  CIVC. 
Specifically,  E&V  technology  should  be  used  to  facilitate  the  PCIS  standardization  activity.  The  PCIS  is 
a  standard  being  developed  in  conjunction  with  our  NATO  allies  that  combines  the  best  features  of  both 
CAIS-A  and  the  equivalent  European  standard  (PCTE+).  The  PCIS  activity  is  planning  to  achieve 
standardization  in  mid- 1994.  The  technology  developed  for  the  CIVC2  contract  would  facilitate 
validation  and  conformance  testing  for  the  PCIS  standard  and  should  be  transitioned  to  this  forthcoming 
standards  activity.  This  technology  involves  actual  tests,  analysis  tools,  and  a  methodology  for  develop¬ 
ment  that  includes: 

•  Information  Models  —  These  models  facilitate  the  design  and  assessment  of  the 
coverage  of  validation  test  suites.  They  also  assist  in  the  prioritization  of  test 
objectives  and  leverage  testing  effoit  more  effectively.  A  framework  for  manage¬ 
ment  of  test  suites  is  also  provided  by  information  models. 

•  Test  Selection  and  Prioritization  —  Test  selection  and  prioritization  involves  the 
development  of  a  strategy  to  select  areas  of  a  standard  for  initial  test  development 
and  other  areas  for  later  test  development.  This  prioritization  activity  is  difficult  and 
error  prone  if  done  in  an  ad  hoc  manner.  The  CIVC  effort  has  developed  methods 
for  the  analysis  of  systems  with  information  models  that  result  in  intelligent 
selection  of  tests  to  develop  for  validation  suites.  Coverage  by  a  test  suite  is  increased 
by  these  methods,  and  the  development  efforts  are  consequently  more  cost  effective. 

•  Test  Case  Generation  —  Semantic  and  behavioral  models  of  interface  sets  may 
enable  the  automated  generation  of  test  cases.  Prototypes  already  developed  by  the 
QVC  contractor  indicate  that  this  approach  is  feasible.  An  automated  approach 
would  enable  a  much  greater  number  of  test  cases  to  be  generated  compared  to  the 
amount  produced  by  a  manual  coding  method. 

•  Hypermedia  Traceability  Systems — The  CIVC  effort  has  developed  hypeimedia- 
based  systems  for  interactive  presentation  of  test  suite  traceability.  These  systems 
significantly  increase  users’  understanding  of  die  derivation  of  software  products. 

In  less  than  ten  years,  the  ACVC  has  more  than  tripled  in  size,  in  response  to  the  need  to  test  prod¬ 
ucts  for  conformance  to  the  language  standard.  This  fact  underscores  the  need  for  validation  capabilities 
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and  indicates  the  importance  of  maintainability  to  decreasing  the  cost  of  such  a  set  of  tests.  A  significant 
portion  of  the  effort  to  develop  a  suite  should  be  devoted  to  easing  enhancement.  In  cases  where  test 
suites  are  developed  using  alternate  approaches,  they  should  be  delivered  in  a  form  such  that  the  standard 
maintenance  process  can  be  used  to  maintain  and  enhance  them. 

Finally,  multiple  implementations  of  a  standard  create  a  situation  where  selections  should  be 
made  using  results  of  evaluation  activities.  Although  there  may  never  be  a  large  number  of  competing 
implementations  of  some  APSE  standards,  evaluation  data  is  important  for  determining  the  suitability  of 
an  implementation  for  a  given  application.  Therefore,  it  is  recommended  that  efforts  to  create  evaluation 
capabilities  for  standardized  APSE  interfaces  be  initiated. 


4.7  PROMOTE  INTERNATIONAL  SHARING  OF  E&V  TECHNOLOGY,  BUT  AVOID 

JOINT  MULTINATIONAL  DEVELOPMENT  OF  ASSESSOR  PRODUCTS 

Sharing  of  technology  internationally  should  be  encouraged  to  the  maximum  extern  possible  to 
avoid  duplication  of  effort  and  to  promote  cooperation  between  nations.  Evaluation  and  validation  tech¬ 
nology  should  be  considered  less  sensitive  as  a  technology  than  other  software  systems,  such  as  actual 
military  applications  or  APSE  tools  used  to  build  military  applications.  The  ACEC  is  basically  the  same 
type  of  technology  as  the  ACV C,  so  any  difference  in  the  export  controls  placed  on  these  two  products  is 
both  confusing  and  counterproductive.  Further,  making  the  suite(s)  available  electronically  would 
enhance  the  user  community’s  ability,  both  at  home  and  abroad,  to  access  and  use  the  most  current  suite  in 
the  most  timely  manner.  Although  the  practice  of  international  development  of  standards  is  well 
established  and  the  international  sharing  of  E&V  technology  should  be  promoted,  joint  international 
development  of  E&V  technology  is  neither  cost  effective  nor  warranted.  In  fact,  the  logistical,  political, 
and  language  barriers  argue  strongly  against  joint  development  of  E&V  technology. 


4-11 


E&V  Team  Final  Report 


5. 


CONCLUSION 


Much  progress  has  been  made  in  the  area  of  Evaluation  and  Validation  since  the  E&V  Team  was 
established  in  1983.  At  its  inception,  this  was  virtually  an  unexplored  Held  of  study.  Now,  a  significant 
amount  of  usable  technology  is  available.  However,  there  is  still  much  work  to  be  done.  As  APSE 
components  mature,  the  areas  of  concern  change .  This  results  in  the  need  for  new  types  of  E&V  technol¬ 
ogy  to  address  the  new  critical  issues.  The  E&V  Team  has  done  its  best  to  make  sure  that  existing  E&V 
technology  is  both  suitable  and  effective  for  application  on  today ’s  programs.  It  has  also  tried  to  identify 
the  needs  of  the  future.  E&V  Project  accomplishments  provide  an  excellent  starting  point  for  continued 
development  of  APSE  E&V  technology. 
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APPENDIX  A 
LIST  OF  ACRONYMS 


ACEC 

ACECWG 

ACM 

ACVC 

AES 

AFWAL 

AJPO 

APSE 

APSEWG 

ARTEWEG 

ASEET 

ATF 

AVO 

CAIS 

CAIS-A 

CAISWG 

CASE 

OVC,  CIVC1 
CIVC2 

avcwG 

CLASSWG 

CRAC 

CVCWG 

DoD 

DR 

E&V 

FFRC 

KAPSE 

KIT 


Ada  Compiler  Evaluation  Capability 

ACEC  Working  Group 

Association  for  Computing  Machinery 

Ada  Compiler  Validation  Capability 

Ada  Evaluation  System  (United  Kingdom  product) 

Air  Force  Wright  Aeronautical  Laboratories  (later  WRDC,  now  WL) 

Ada  Joint  Program  Office 

Ada  Programming  Support  Environment 

APSE  Working  Group  (proceeded  CLASSWG) 

Ada  Run  lime  Environments  Working  Group  (of  SIGAda) 

Ada  Software  Engineering  Education  and  Training 

Advanced  Tactical  Fighter 

Ada  Validation  Office 

Common  APSE  Interface  Set 

CAIS  (Version  A) 

CAIS  Working  Group  (proceeded  SEVWG) 

Computer  Aided  Software  Engineering 

CAIS  Implementation  Validation  Capability 

OVC  for  CAIS-A 

CTVC  Working  Group 

Classification  Working  Group 

Computer  Resources  Acquisition  Course 

CAIS  Validation  Capabilities  Working  Group  (early  CIVCWG) 

Department  of  Defense 

Distinguished  Reviewer 

Evaluation  and  Validation 

Federally  Funded  Research  Center 

Kernel  APSE 

KAPSE  Interface  Team 
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NATO  North  Atlantic  Treaty  Organization 

PCIS  Portable  Common  Interface  Set 

PCTE+  Ponable  Common  Tool  Environment 

PIWG  Performance  Issues  Working  Group  (of  SIGAda) 

PTS  Performance  Testing  Service 

PUBWG  Publicity  Working  Group 

PAG  Procedures  and  Guidelines 

QTS  Quality  Testing  Service 

REQWG  Requirements  Working  Group 

R&D  Research  and  Development 

RFP  Request  for  Proposal 

RTE  Run  Time  Environment 

SDI  Strategic  Defense  Initiative 

SEI  Software  Engineering  Institute 

SEVWG  Standards  Evaluation  and  Validation  Working  Group 

SIGAda  Special  Interest  Group:  Ada  (of  the  ACM) 

STARS  Software  Technology  for  Adaptable  Reliable  Systems 

TASC  The  Analytic  Sciences  Corporation 

USAF  United  States  Air  Force 

VDD  Version  Description  Document 

WG  Working  Group 

WL  Wright  Laboratory  (formerly  WRDC) 

WRDC  Wright  Research  and  Development  Center  (formerly  AFWAL,  now  WL) 

WPAFB  Wright  Patterson  Air  Force  Base 
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APPENDIX  B 

E&V  TEAM  PARTICIPANTS 


The  following  list  was  culled  from  E&V  Team  Public  Reports  and  Meeting  Minutes.  Names  of 
guest  speakers  and  those  who  participated  only  briefly  have  been  omitted.  Everyone  who  participated  in 
any  way,  however  briefly,  is  thanked  for  their  support  and  we  apologize  to  any  significant  contributors 
whose  names  were  inadvertently  omitted.  The  names  marked  with  asterisks  are  those  who  participated 
significantly  during  the  final  phase  and  helped  formulate  the  team  recommendations  given  in  Section  4. 


NAME 

*  Abraham,  Rebecca  Capt. 

*  Adams,  Karyl 
♦Ashby,  Sam 

Anderson,  Christine 


ORGANIZATION 

WRDC/FIOC,  WPAFB 
c.j.  kemp  systems 
Boeing  Military  Airplane 
AFATL,  Eglin  AFB 


Bailey,  Betsy  Kruesi 
♦Brashear,  Philip 
Brookshire,  Jerry 
♦Burlakoff,  Mike 
Bums,  Greg 
Burton,  Dan 


Institute  for  Defense  Analyses 
CTA  Incorporated 
Texas  Instruments 
SW  Missouri  State  University 
GTE/Govemment  Systems 
ESD/ALL,  Hanscom  AFB 


Camp,  John 
Castor,  Virginia 
♦Chilson,  Lynn 
Chitwood,  Georgeanne 
♦Clark,  Peter 
♦Crawford,  Baal 


WRDC/AAAF-2,  WPAFB 
AJPO,  Pentagon 
SofTech 


ASD/ADOL,  WPAFB 

TASC 

TASC 


Deese,  Albert  Capt. 
Dobbs,  Paul 


ASD/SIOL  WPAFB 
General  Dynamics 


*Eilers,  Dan 
Elderhorst,  Linda 
Estes,  Nelson 


Irvine  Compiler  Coip. 

Sys.  Eng.  Test  Direct.  Patuxent  River,  MD 
ASD/AFALC/AXTS,  WPAFB 


Facemire,  Jeff 
Fainter,  Robert 
Fanning,  Shawn 
♦Ferguson,  Clarence  “Jay” 
Fleming,  Richard 


SofTech 

Arizona  State  University 
SofTech 

National  Security  Agency 
The  Aerospace  Corp. 
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NAME 

ORGANIZATION 

1 

*Foidl,  Jack 

TRW,  Systems  Division 

■ 

♦Francl,  Fred 

Sonicraft,  Inc. 

1 

Fritz,  Robert 

Computer  Sciences  Corp. 

*Gicca,  Greg 

Sanders 

1 

Gilroy,  Kathleen 

Software  Productivity  Solutions 

Gralia,  Mars 

APL,  Johns  Hopkins  University 

■ 

♦Gutzmann,  Kurt 

SofTech 

1 

*Hackett,  Kevin 

SofTech 

Hammons,  Charles 

Texas  Instruments 

I 

Hanna,  Bruce  Capt. 

WRDC/AAAF-2,  WPAFB 

■ 

Harto,  Debra 

AFATL/DLCM,  Eglin  AFB 

♦Hazle,  Marlene 

MITRE 

I 

Henne,  Marlow 

Harris  Corp. 

■ 

Holmes,  Tracy 

GTE  Government  Systems 

Humphrey,  Terry 

Johnson  Space  Center 

1 

♦Impicciche,  Alan 

Naval  Avionics  Center 

■ 

*  Jennings,  Donald 

OC-ALC/MMECO,  Tinker  AFB 

1 

♦Kean,  Elizabeth 

RADC/COEE,  Griffiss  AFB 

1 

Kirkbride,  Kathy 

Oneida  Resources 

Kirkpatrick,  James  Lt. 

AFALC/PTEC,  WPAFB 

Kopp,  Allan  Maj. 

AJPO,  Pentagon 

■ 

Kramer,  Jack 

Institute  for  Defense  Analyses 

■ 

LaPointe,  Mike  Lt 

AD/ENE,  Eglin  AFB 

■ 

♦Lawlis,  Patricia  Maj. 

AFIT,  WPAFB 

1 

♦Leavitt,  Thomas 

Boenig  Military  Airplane 

Leavitt,  Randal 

PRIOR  Data  Science 

| 

LeGrand,  Sue 

SofTech 

■ 

♦Lindquist,  Tim 

Arizona  State  University 

Long,  R.  Lt. 

WRDC/AAAF-2,  WPAFB 

I 

*  Maher,  Patrick 

Motorola  GEG 

Mark,  Donald 

RADC/COEE,  Griffiss  AFB 

■ 

Marmelstein,  Robert  Lt. 

WRDC/AAAF-2,  WPAFB 

1 

♦Martin,  Ronnie 

Purdue  University 

McBride,  John 

SofTech 

1 

■ 
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NAME 

*  McKee,  Gary 
Meirink,  Michael 
Miller,  John 
Mills,  Michael 

*  Mulholland,  Sandi 
Munun,  Hans 
Munck,  Robert 
Myers,  Philip  LCDR 

Obemdorf,  Patricia 
Olson,  Douglas  Lt. 

Pickart,  Bruce  Capt. 
Probert,  Tom 

Reddan, John 
Reilly,  Paul 

*  Remkes,  David 
Rhoads,  Barbara 
Rhoden,  Victoria  Capt. 
Rohrer,  Amos 
Romanowski,  Helen 

Sabota,  Darlene  Lt. 
Sandborgh,  Raymond 
Schaar,  Brian 
Stacey,  Christine 
Stanton,  John 

*  Stiles,  Lloyd 

*  Szymanski,  Raymond 

Taylor,  Guy 

*  Terrell,  Kermit 

*  Thomas,  Jerry 
Tompkins,  Mary  Aim 

*  Weiderman,  Nelson 
Williamson,  James 

*  Wills,  Betty 
Witt,  Donald 


ORGANIZATION 

MeKee  Consulting 
Unisys 

SM-ALC/MMEHD,  McClellan  AFB 
ASD-AFALC/AXTS,  WPAFB 
Rockwell  International 
Naval  Ocean  Systems  Center 
Unisys 

AJPO,  Pentagon 

Naval  Ocean  Systems  Center 
HQ  AFCMD/SID,  Kirtland  AFB 

HQ  AFOTEC/LG5S 
Institute  for  Defense  Analyses 

SYSCON  Corporation 
Data  General  Corporation 
SofTech 

Oneida  Resources 
ASD/TASE,  WPAFB 
EG&G 

Rockwell  International 

AFWAL-FIGRB ,  WPAFB 
Unisys 

AJPO,  Pentagon 
GTE  Government  Systems 
AJPO,  Pentagon 
FCDSSA,  San  Diego 
WL/AAAF-3,  WPAFB 

FCDSSA,  Dam  Neck,  VA 
Boeing  Military  Airplane 
Naval  Ocean  Systems  Center 
Lockheed  Missiles  &  Space  Co. 

Software  Engineering  Institute 
WRDC/AAAF-2,  WPAFB 
CCSO/XPTB,  Tinker  AFB 
AFTT/EN,  WPAFB 


B-3 


E&V  Team  Final  Report 


APPENDIX  C 
REFERENCES 


[ACEC 1990]  “Ada  Compiler  Evaluation  Capability  (ACEC)  Technical  Operating  Report  (TOR)  Read¬ 
er’s  Guide,’’  Air  Force  Wright  Research  and  Development  Center,  D500-1247-1,  May  1990. 

[ACEC  1990a]  “Ada  Compiler  Evaluation  Capability  (ACEC)  Technical  Operating  Report  (TOR) 
User’s  Guide,”  Air  Force  Wright  Research  and  Development  Center,  D500- 12470-1,  May  1990. 

[ACEC  1990b]  “Ada  Compiler  Evaluation  Capability  (ACEC)  Technical  Operating  Report  (TOR)  Ver¬ 
sion  Description  Document,”  Air  Force  Wright  Research  and  Development  Center,  D500- 12472-1, 
May  1990. 

[Brooks  1987]  Brooks,  FP.  “No  Silver  Bullet  —  Essence  and  Accidents  of  Software  Engineering,” 
Computer,  pp.  10-19,  April  1987. 

[CIVC  1989]“OVC  Implementor’s  Guide,”  SofTech,  Inc.  OVC-FINL-19-1, 16  October  1989. 

[CTVC  1990]  “OVCl  Framework,”  SofTech,  Inc.  CVC-VREL-2/1-01, 1  March  1990. 

[CTVC  1990a]  “Test  Report  Reader’s  Guide  with  Appendix  1  —  Operator’s  Guide,”  SofTech,  Inc. 
CTVTL-FINL-020-02,  19  March  1990. 

[CTVC  1990b]  “Beta  Test  Operator’s  Guide,”  SofTech,  Inc.,  CTV C-FNL-20-03, 18  July  1990. 

[CVP 1985]  “Component  Validation  Procedures  (CVP)  Document,”  APSE  E&V  Team  CAISWG,  1  De¬ 
cember  1985. 

[DoD  1983]  ANSI/MEL-STD- 1 8 1 5  A- 1 983 ,  “Reference  Manual  for  die  Ada  Programming  Language,” 
U.S.  Department  of  Defense,  17  February  1983,  DTTC  Number  AD  A131  511. 

[DoD  1986]  DoD-STD-1838,  “Common  APSE  Interface  Set  (CAIS),”  U.S.  Department  of  Defense,  9 
October  1986,  DTIC  Number  AD  A157  589. 

[DoD  1988]  DoD-STD-2167A,  “Defense  System  Software  Development,”  U.S.  Department  of  De¬ 
fense,  29  February  1988. 

[DoD  1989]  MIL-STD-1838A,  “Common  APSE  Interface  Set,  Revision  A,”  U.S.  Department  of  De¬ 
fense,  April  19898,  DTIC  Number  AD  A157  589. 

[E&V  GB  1989]  “E&V  Guidebook,  Version  2.0,”  Wright  Research  and  Development  Center,  30  Sep¬ 
tember  1989,  DTIC  Number  AD  A214  166. 

[E&V  Report  1984]  Castor,  V.,  “Evaluation  and  Validation  (E&V)  Team  Public  Report,  Volume  1,”  Air 
Force  Wright  Aeronautical  Laboratories,  Wright-Patterson  AFB,  30  November  1984,  DTIC  Number 
AD  A153  609. 

[E&V  Report  1985]  Szymanski,  R.,  “Evaluation  and  Validation  (E&V)  Team  Public  Report,  Volume 
II,”  Air  Force  Wright  Aeronautical  Laboratories,  Wright-Patterson  AFB,  November  1985,  DTIC  Num¬ 
ber  AD  A 172  343. 


C-l 


1 


E&V  Team  Final  Report 


[E&V  Report  1987]  Szymanski,  R.,  “Evaluation  and  Validation  (E&V)  Team  Public  Report,  Volume 
HI,”  Air  Force  Wright  Aeronautical  Laboratories,  Wright-Patterson  AFB,  September  1987,  DTIC  Num¬ 
ber  AD  A196  164. 

[E&V  Report  1989]  Szymanski,  R.,  “Evaluation  and  Validation  (E&V)  Team  Public  Report,  Volume 
IV,”  Air  Force  Wright  Aeronautical  Laboratories,  Wright-Patterson  AFB,  January  1989,  DTIC  Number 
AD  A206  394. 

[E&V  Report  1990]  Szymanski,  R.,  “Evaluation  and  Validation  (E&V)  Team  Public  Report,  Volume  V,” 
Air  Force  Wright  Aeronautical  Laboratories,  Wright-Patterson  AFB ,  December  1 990,  DTIC  Number  to 
be  assigned. 

[E&V  RM 1989]  “E&V  Reference  Manual,  Version  2.0,”  Wright  Research  and  Development  Center,  30 
September  1989,  DTIC  Number  AD  A214  167. 

[E&V  Requirements  1984]  “E&V  Requirements  Document,”  REQWG,  included  in  [E&V  Report 
1984]. 

[E&V  Requirements  1987]  “E&V  Requirements  Document,  Revised,”  REQWG,  included  in  [E&V 
Report  1987]. 

[E&V  Tools  and  Aids  1990]  “E&V  Tools  and  Aids  Document,”  REQWG,  included  in  [E&V  Report 
1990]. 

[E&V  Workshop  1984]  V.L.  Castor,  et  al,  “Ada  Programming  Support  Environment  (APSE)  Evaluation 
and  Validation  (E&V)  Workshop  Report,”  Institute  for  Defense  Analysis,  December  1984,  DTIC  Num¬ 
ber  AD  A156  667. 

[Issues  1987]  “Issues  and  Strategies  for  CAIS  Evaluation  and  Validation,  Version  1.0,”  APSE  E&V 
Team  SEVWG,  September  1987,  in  [E&V  Report  1987]. 

(Issues  1990]  “Issues  and  Strategies  for  Evaluation  and  Validation  of  CAIS -A  Implementations,”  based 
on  April  6, 1989  MEL-STD-1838-A,  APSE  E&V  Team  SEVWG,  February  1990. 


C-2 


